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ABSTRACT 

The objective of this paper is to propose a novel approach to enhance working memory (WM) training for mobile devices 
by using information about the arousal level of a person. By the example of an adaptive n-back task, we combine 
methodologies from different disciplines to tackle this challenge: mobile learning, affective computing and cognitive 
psychology. Mobile applications for WM training offer the advantage that a user can perform a training session 
independently of time and location. Using approaches from affective computing, training sessions can be made more 
challenging and engaging by making them adaptive to the current state of a user. In contrast to conventional adaptive 
WM trainings, our approach uses physiological signals to extract information about the arousal level of a person. In this 
paper, we address the question how information about the arousal level of a user can be captured and how this 
information can be used to influence the overall learning procedure. An approach to integrate feedback about the arousal 
level of a person into mobile WM training is presented and exemplified using the «-back task. 
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1. INTRODUCTION 

Within the last few years there has been a strong tendency of learning applications getting mobile. This 
requires rethinking of how learning applications are designed as learning performance may differ due to 
varying environmental conditions which cause the user to be in different affective states. Therefore, it is 
important to integrate affective information into mobile learning applications. 

The concept of integrating information about user affect into learning environments is not new; it was 
first introduced by Picard (1995). Since then, much research has been done towards learning systems which 
consider affective information in a stationary learning environment (e.g. D'Mello et al., 2007). 

We present an approach which uses innovative wearable sensor technologies integrated into a mobile 
learning environment. Using the example of the u-back task, we illustrate the practical use of such an 
affective mobile WM training and how it can help to improve the overall learning process. Not only can it 
help to get a more differentiated picture of the learning process but also to prevent users from being bored or 
frustrated due to too easy or too difficult tasks. Especially in a mobile environment, information about the 
emotional arousal of a user can help to incorporate changing environmental conditions. For instance, the 
difficulty at the beginning of the task can be adjusted to the current arousal level. Moreover, results of a task 
can be presented in relation to prior trainings where a user has been in a similar state. 

This paper is organized as follows: section 2 describes the theoretical background of WM training and 
how physiological signals can be used to gather information about the arousal level of a person. After 
specifying the research questions in section 3, section 4 describes a research approach which integrates 
information about emotional arousal into a mobile WM training. In section 5, we describe how the proposed 
system can help to improve mobile learning experience. 
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2. THEORETICAL BACKGROUND 

The term working memory refers to the ability to temporary store and process information for complex 
cognitive tasks at the same time (Baddeley, 1992). WM represents the ability to control attention (e.g. Engle, 
2002) and its capacity is correlated with a large variety of higher-order cognitive tasks such as reading 
comprehension, complex learning and reasoning (Daneman & Carpenter, 1980). WM capacity decreases 
constantly with aging (e.g. Hale et al., 2011). Continuous training of WM can help to maintain cognitive 
abilities for a longer time. For younger adults, training of WM can help to improve performance in many 
areas. As a result, a large number of WM tasks and trainings have been developed. During WM tasks, 
information has to be kept in active memory while distracting or interfering activities are performed. Recent 
studies indicate that WM training which adapts the difficulty to the learning progress is more effective than 
training which does not adapt to the current state of a user (Klingberg et al., 2005; Schmiedek et al., 2010). 
However, these trainings adapt solely based on prior performance in the training task. The affective state of a 
person remains disregarded although the engagement of a person in the learning process has an important 
influence on the learning outcome. Presenting learning materials in an activating way keeps a person 
motivated and can help to maximize learning outcome (Dior, 2008). 

Emotion and activation are accompanied by physiological arousal which is reflected in reactions like 
increased heart rate or skin conductance (Grandey, 2000). According to the Yerkes-Dodson-Law there exists 
an optimal arousal level for a high learning performance. While a very high arousal level can be 
advantageous in simple tasks, for difficult tasks it can cause a decrease in performance (Yerkes & Dodson, 
1908). Therefore, it is crucial to detect the optimal arousal level during a task to achieve the best learning 
performance. Questionnaires and interviews are commonly used for this purpose. However, these traditional 
methods may fail when used in a mobile learning environment. For instance, Robinson & Clore (2002) 
found, that if the presentation time of a questionnaire is too far away from an event, answers underlie 
systematic biases. Moreover, data from questionnaires and interviews might not be honest and be influenced 
by subjective factors (Wali et al., 2009). In contrast to these methods, using physiological data can provide an 
objective measure of the affective state of a person as physiological signals are controlled by the central 
nervous system and are relatively resistant to manipulation. Additionally, physiological signals are constantly 
emitted which allows continuous monitoring of the user state. However, in mobile learning environments, 
physiological signals have to be recorded in an unobtrusive way that does not affect a person in his or her 
daily routine. New wearable sensor technologies can help towards this goal. Morris & Aguilera (2012) 
subsume the technical development of the last years: “Much of the wearable and environmental sensing that 
was considered futuristic only five years ago have become practical or at least easily imaginable tools for 
daily life”. This opens up new possibilities for ambulatory assessment of physiological data. 

There has been a lot of research towards systems that are able to recognize arousal from physiological 
data. For instance Fichtenstein et al. (2008) achieved a recognition rate of 82% on a 5 point arousal scale 
using electrocardiogram (ECG), electrodermal activity (EDA), breathing rate, temperature and 
electromyographic (EMG) signals. Haag et al. (2004) achieved even higher recognition rates (89.73% on a 
continuous arousal scale with a bandwidth of 10%) with a neural network classifier using features extracted 
from ECG, EDA, skin temperature, blood volume pulse, EMG and respiration. In the following, we describe 
how arousal recognition from physiological signals can be combined with mobile learning applications. 


3. RESEARCH QUESTIONS 

As mentioned above, so far the adaption of the difficulty of WM tasks has solely been done based on prior 
task performance. The current state of a user remaines disregarded. Using information about a person's 
arousal level to adapt task difficulty of a mobile WM training can bring major advantages: 

• At the beginning of the training, task difficulty can be adapted to the current arousal level. Hence, 
training always starts at a challenging but not overstraining level and is therefore more effective. 

• During the training, task difficulty can be decreased before a user is frustrated (e.g. due to high error 
rates) and increased before a user gets bored due to mental underload. 

• Training results can be presented in relation to previous results when the user was in a similar state like 
at the time of the training. 
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The integration of arousal information into a mobile WM application raises the following research 
questions (RQ): 

RQ1: Which physiological sensors are suited best in order to unobtrusively derive feedback about the 
arousal level of a person in a mobile learning scenario? 

The major requirements for the selection of physiological signals are: (a) it has to be possible to record 
the signals continuously over a longer period in an unobtrusive manner, (b) the signal is highly corre-lated 
with arousal and (c) the signal quality has to be sufficient to extract arousal information online. 

RQ2: How can the information about the arousal level be integrated in a mobile learning application? 

The arousal information which is extracted from the sensor data has to be provided to the mobile learning 
application. A flexible solution which allows to easily exchange sensors or algorithms in the future is 
preferred for this purpose. 

RQ3: Which mechanisms can be developed to adapt the difficulty of a WM training? 

For the WM training it is crucial to find appropriate mechanisms how the training has to adapt according 
to different levels of arousal. 

RQ4: Can the learning outcomes really be improved by affective feedback? 

Integrating information about the arousal level derived from physiological signals into a mobile WM 
application is a novel approach. Thus, it has to be validated, how far this approach can help to increase 
effectiveness of the WM training. 


4. RESEARCH APPROACH 

In the following, we describe how physiological sensors can be used to extract information about the arousal 
level of a person and how this information can be integrated into a mobile WM training. Moreover, we 
illustrate how the WM application can adapt to the arousal level of a person and how effectiveness of such an 
application can be validated. 

RQ1: ECG, EDA and eyeblink rate (EBR) are well known to correlate with the arousal level of a person 
and can be recorded noninvasively with unobtrusive wearable devices. Therefore, we decided to use these 
signals for our learning system to obtain information about the arousal level. Heart rate - derived from the 
ECG signal - reflects the activity of the sympathetic and the parasympathetic nervous system. While 
emotional arousal and stress cause heart rate to increase, relaxation decreases it (Berntson et al., 2007). EDA 
is a well-established parameter in psychophysiological studies about arousal. It has for instance been used for 
arousal classification by Healey & Picard (2005) and Reinhardt et al. (2012). EBR is also influenced by 
cognitive processes such as arousal. For instance. Oh et al. (2012) found an increase in EBR during the 
Stroop task. Moreover, Bentivoglio et al. (1997) found that EBR significantly increased during conversation 
while it decreased during reading. To record ECG and EDA we use the wearable ECG chest belt ekgMove 
and the EDA bracelet edaMove developed by movisens. Both sensors have been developed such that they 
can be worn over a longer period without disturbing a person in his or her daily routine. Moreover, the 
sensors can be placed such that the signal is relatively robust to movement artifacts. To gather information 
about EBR, we use the integrated camera of the mobile device on which the WM training is performed. 
Systems like EyeGuardian (Han et al., 2012) and EyePhone (Miluzzo et al., 2010) have successfully 
demonstrated that it is possible to detect eye blinks using the integrated camera of mobile devices. The 
algorithms used to extract information about the arousal level have to be very light-weight and have to be 
able to deal with small window-sizes as the results have to be provided online on the mobile device. 

Preliminary results of a study where high and low arousal were induced in a laboratory setting indicate, 
that heart rate, heart rate variability and EDA amplitude are very well suited to differentiate between arousal 
levels. 

RQ2: Integrating information about arousal into the affective learning application in real-time requires 
software which is able to extract the arousal information from the raw sensor data and to provide it to the 
learning application. Moreover, it should be possible to easily exchange components without making changes 
to the rest of the framework as new sensor devices and algorithms might become available in the near future. 
Therefore, we decided to use the xAffect software (Schaaff et al., 2012). xAffect is a modular middleware 
which has been developed as a solution to compute online biofeedback from physiological sensor data. A 
large number of components can be combined in order to obtain the desired setup for a specific application. 
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One great advantage of the modular design of the xAffect middleware is that it allows rapid prototyping. If 
new innovative sensor devices become available or new algorithms for arousal recognition are developed, the 
components currently implemented can easily be replaced or extended by the new components without 
making changes to the rest of the framework. For the current research, the ekgMove and the edaMove sensor 
as well as the integrated camera of the mobile device are integrated as data sources. Appropriate data 
processors to classify the arousal level of a person have to be implemented based on previous findings about 
arousal classification from other studies. The arousal level is computed by xAffect as a combination of 
features from all data sources. All recorded data can be logged for later analysis. 



Signal Feature extraction Difficulty 

recording & classification adaption 


Figure 1 . Design of the affective learning system 

The xAffect software is integrated into the mobile learning application as a library. Figure 1 illustrates the 
elements of the affective mobile learning framework and their interplay. 

RQ3: According to the arousal information, which the learning application receives from the xAffect 
software, the difficulty of the learning application can be adapted to the current arousal level of a person. It is 
important that the difficulty is always kept on a challenging but not overstraining level. There is a large 
variety of tasks for WM training. To make WM training adaptive, appropriate mechanisms have to be found. 
We use the n-back task as an example to demonstrate how WM training can be made adaptive to the arousal 
level of a user. In the n-back-task a sequence of stimuli is presented to the user and the user has to decide 
whether a stimulus matches the stimulus n steps earlier. There are several factors in the n-back task which 
can be used to vary task difficulty: presentation time, interstimulus interval and the number of items or the 
dimensionality of items for the spatial n-back task respectively. At the beginning of a training session the 
current arousal level of a person is used to determine the optimal complexity of how to start the training. 

Currently, the adaptive n-back task is implemented as a desktop version. For the mobile WM training it 
will be transferred to a mobile system. 

RQ4: The integration of arousal information into a mobile WM training is a novel approach. Thus, the 
effectiveness of the affective WM training has to be evaluated. For this purpose, the n-back task will be 
implemented in two versions: an adaptive version and a non-adaptive version which can be used to train a 
control group. In the validation study, the treatment group and the control group will use the WM training on 
a daily basis over a longer period of time. The study will be accompanied by questionnaires using a mobile 
experience sampling system. A second objective of the study will be to find out if motivation of participants 
is higher if changes in performance are computed in relation to training days where a person’s arousal level 
has been similar to the one in a previous session. As WM performance varies systematically across days 
(Brose et al., 2012), this can help to obtain a more distinguished picture of the training success. 


5. EXPECTED OUTCOME 

With the current research we present an approach how affective information can be integrated into mobile 
learning devices. Methods from biosignal processing and affective computing are combined with a mobile 
learning application for WM training. The application will be able to adapt task difficulty based on the 
arousal level of a user. The arousal information will be extracted from physiological sensor data. We 
hypothesize that using this system the performance in WM tasks can be significantly improved in comparison 
to existing approaches as WM training is most effective when done at a challenging level. One limitation of 
the system is that additional sensors have to be worn to measure ECG and EDA. Future works will therefore 
include research towards even less obtrusive sensors to capture data. Moreover, additional environmental 
parameters captured by the mobile device - such as the current location of a user or the background noise - 
can be integrated in the adaptive process. Modulating the training difficulty to keep the arousal at an optimal 
level by using task and biofeedback based adaption mechanisms will help to maximize the learning outcome. 
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